Sample-Efficient Iterative Lower Bound Optimization of Deep Reactive Policies for Planning in Continuous MDPs

Recent advances in deep learning have enabled optimization of reactive policies (DRPs) for continuous MDP planning by encoding a parametric policy as neural network and exploiting automatic differentiation an end-to-end model-based gradient descent framework. This approach has proven effective optimizing DRPs nonlinear MDPs, but it requires large number sampled trajectories to learn effectively can suffer from high variance solution quality. In this work, we revisit the overall DRP objective instead take minorization-maximization perspective iteratively optimize w.r.t. locally tight lower-bounded objective. novel formulation iterative lower bound (ILBO) is particularly appealing because (i) each step structurally easier than objective, (ii) guarantees monotonically improving under certain theoretical conditions, (iii) reuses samples between iterations thus lowering sample complexity. Empirical evaluation confirms that ILBO significantly more sample-efficient state-of-the-art planner consistently produces better quality with variance. We additionally demonstrate generalizes well new problem instances (i.e., different initial states) without requiring retraining.